Medical image processing apparatus, medical image processing method, and medical observation system

ABSTRACT

A medical observation system including a medical imaging device that captures a plurality of images of a living body while changing a focus position, and circuitry that generates a composite image by compositing the plurality of images captured by the medical imaging device, and switches output between the generated composite image and one of the plurality of images based on a result of analysis performed on at least one of the plurality of images.

TECHNICAL FIELD

The present technology relates to a medical image processing apparatus,a medical image processing method and a medical observation system,particularly to a medical image processing apparatus, a medical imageprocessing method and a medical observation system by which a deep focusimage can be obtained with a low delay and at a high frame rate, forexample.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Japanese Priority PatentApplication JP 2015-124005 filed on Jun. 19, 2015, the entire contentsof which are incorporated herein by reference.

BACKGROUND ART

A depth of field of a photographed image obtained by a medicalmicroscope or the like is shallow, where an image of an object at aslightly different depth from a focus position (focus plane) that is infocus in a real space is blurred.

In a photographed image obtained by the medical microscope in a brainsurgery, for example, a frontward object is photographed in theperiphery while an object (such as a surgical site) being a focus ofattention located more or less at the back is photographed at thecenter. At this time, the object in the periphery of the photographedimage is blurred when a focus is adjusted to bring the object at thecenter of the photographed image into focus, which may affectobservability of the object and operability of the medical microscope.

Accordingly, for example, there is proposed a real-time all-in-focusmicroscopic camera which performs high-speed photographing of an imagewhile changing a focus position (focal length) and obtains a deep focusimage (an all-in-focus image) from a plurality of images obtained by thephotographing (refer to Patent Literature 1, for example).

CITATION LIST Patent Literature

PTL 1: International Publication No. WO 2002/082805

SUMMARY OF INVENTION Technical Problem

Now, when an image is to be provided to a user such as a doctorperforming an operation with medical equipment such as the medicalmicroscope, it is desirable from the nature of the medical field thatthe image is provided with a low delay and at a high frame rate.

In view of such circumstances, it is desirable to be able to obtain adeep focus image with a low delay and at a high frame rate.

Solution to Problem

A medical image processing apparatus according to an embodiment of thepresent technology includes circuitry configured to generate a compositeimage by compositing a plurality of images obtained by capturing with amedical imaging device a living body while changing a focus position,and switch output between the generated composite image and one of theplurality of images based on a result of analysis performed on at leastone of the plurality of images.

A medical image processing method according to an embodiment of thepresent technology includes generating a composite image by compositinga plurality of images obtained by capturing with a medical imagingdevice a living body while changing a focus position, and switchingoutput between the generated composite image and one of the plurality ofimages based on a result of analysis performed on at least one of theplurality of images

A medical observation system according to an embodiment of the presenttechnology includes a medical imaging device configured to capture aplurality of images of a living body while changing a focus position,and circuitry configured to generate a composite image by compositingthe plurality of images captured by the medical imaging device, andswitch output between the generated composite image and one of theplurality of images based on a result of analysis performed on at leastone of the plurality of images.

Note that the medical image processing apparatus and the medicalobservation system may be independent apparatus and system or aninternal block making up a single apparatus.

Advantageous Effects of Invention

According to an embodiment of the preset technology, the deep focusimage can be obtained with the low delay and at the high frame rate, forexample.

Note that the effect is not limited to the one described above but maybe any effect described in the present disclosure.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating a configuration example of anembodiment of a medical observation system to which the presenttechnology is applied.

FIG. 2 is a block diagram illustrating a first configuration example ofa signal processing device 12.

FIG. 3 is a diagram illustrating an overview of photographing performedto obtain a photographed image in a photographing unit 11 and generatinga composite image in the signal processing device 12.

FIG. 4 is a diagram illustrating an example of processing performed byan alignment unit 32 and a composition unit 33.

FIG. 5 is a diagram illustrating a photographed image that isphotographed while changing a focus position in the photographing unit11.

FIG. 6 is a flowchart illustrating a first operational example of themedical observation system.

FIG. 7 is a flowchart illustrating a second operational example of themedical observation system.

FIG. 8 is a flowchart illustrating a third operational example of themedical observation system.

FIG. 9 is a block diagram illustrating a second configuration example ofthe signal processing device 12.

FIG. 10 is a diagram illustrating an example of setting a focus shiftrange in a range setting unit 62.

FIG. 11 is a flowchart illustrating a fourth operational example of themedical observation system.

FIG. 12 is a block diagram illustrating a third configuration example ofthe signal processing device 12.

FIG. 13 is a graph illustrating an example of a relationship between afocus position and a focus score.

FIG. 14 is a flowchart illustrating a fifth operational example of themedical observation system.

FIG. 15 is a block diagram illustrating a fourth configuration exampleof the signal processing device 12.

FIG. 16 is a flowchart illustrating an example of processing that setsan AF frame in the signal processing device 12.

FIG. 17 is a block diagram illustrating a configuration example of anembodiment of a computer to which the present technology is applied.

DESCRIPTION OF EMBODIMENTS

<Embodiment of Medical Observation System to which Present Technology isApplied>

FIG. 1 is a block diagram illustrating a configuration example of anembodiment of a medical observation system to which the presenttechnology is applied.

The medical observation system illustrated in FIG. 1 can be applied tomedical equipment such as a medical endoscope system or a medicalelectron microscope (surgical microscope) that has a function ofobserving a living body.

As illustrated in FIG. 1, the medical observation system includes aphotographing unit 11, a signal processing device 12, and a displaydevice 13.

The photographing unit 11 illuminates and photographs an object being aliving body such as a surgical site of a human body subjected to anoperation, for example, and supplies a photographed image being an imageof the living body obtained by the photographing to the signalprocessing device 12.

The photographing unit 11 includes a light source 21, an optical system22 and an image sensor 23.

The light source 21 is formed of a Light Emitting Diode (LED) or thelike and emits light illuminating the object.

The optical system 22 is provided in a lens barrel (not shown) andformed of optical components such as a focus lens and a diaphragm. Theoptical system 22 condenses object light (reflected light) that is thelight emitted from the light source 21, reflected off of the object andincident on the optical system onto the image sensor 23.

The image sensor 23 is a Complementary Metal Oxide Semiconductor (CMOS)sensor, for example, which receives the object light from the opticalsystem 22, performs photoelectric conversion and photographs the object.A photographed image of the object photographed by the image sensor 23is supplied to the signal processing device 12.

Note that the photographing unit 11 can photograph, as the photographedimage, a two-dimension (2D) image and a 3D image formed of a left eyeimage (Left (L) image) and a right eye image (Right (R) image).

When photographing the 3D image, the photographing unit 11 is providedwith the optical system 22 and the image sensor 23 used to photographthe L image, and the optical system 22 and the image sensor 23 used tophotograph the R image as indicated with a dotted line in the drawing.

Moreover, when the photographing unit 11 photographs the 3D image, thesignal processing device 12 performs similar processing on each of the Limage and the R image, for example.

In order to simplify description, it is hereinafter assumed that thephotographing unit 11 photographs the 2D image as the photographedimage.

The signal processing device 12 performs appropriate signal processingon the photographed image obtained from the photographing unit 11 andsupplies an image obtained as a result of the signal processing to thedisplay device 13.

In addition, the signal processing device 12 controls the photographingunit 11 as appropriate.

Specifically, the signal processing device 12 controls the light source21 to control intensity of the illumination provided by the light source21, for example. The signal processing device 12 also controls theoptical system 22 to adjust the diaphragm, a focus (position) and azoom, for example. Moreover, the signal processing device 12 controlsthe image sensor 23 to control a frame rate of a photographed image andexposure time (shutter speed) in photographing to obtain a photographedimage, for example.

The display device 13 displays the image supplied from the signalprocessing device 12. The display device 13 can be a display integralwith the signal processing device 12, a stationary display providedseparately from the signal processing device 12, or a head mountdisplay, for example.

<First Configuration Example of Signal Processing Device 12>

FIG. 2 is a block diagram illustrating a first configuration example ofthe signal processing device 12 in FIG. 1.

As illustrated in FIG. 2, the signal processing device 12 includes aframe buffer 31, an alignment unit 32, a composition unit 33, a drivecontrol unit 34 and a control unit 35.

A photographed image from the photographing unit 11 (specifically theimage sensor 23 thereof) and a composite image (to be described) fromthe composition unit 33 are supplied to the frame buffer 31.

The frame buffer 31 temporarily stores the photographed image from thephotographing unit 11 and the composite image from the composition unit33.

Here, the photographing unit 11 performs (high-speed) photographing at aframe rate higher than or equal to a frame rate of the image displayedin the display device 13 to obtain the photographed image while changinga focus position with the optical system 22 controlled by the drivecontrol unit 34 to be described.

As a result, a plurality of photographed images with different focuspositions is supplied from the photographing unit 11 to the frame buffer31, which stores these plurality of photographed images with differentfocus positions.

The alignment unit 32 performs alignment between the last compositeimage and the latest photographed image stored in the frame buffer 31and supplies the aligned composite image and photographed image to thecomposition unit 33.

Specifically, the alignment unit 32 includes an angle-of-view adjustmentunit 41, a motion blur elimination unit 42, and an object alignment unit43.

The angle-of-view adjustment unit 41 adjusts an angle of view of each ofthe composite image and the photographed image, and supplies thecomposite image and the photographed image after adjusting the angle ofview thereof to the motion blur elimination unit 42.

The motion blur elimination unit 42 eliminates motion blur in thephotographed image supplied from the angle-of-view adjustment unit 41,and supplies the photographed image from which the motion blur iseliminated to the object alignment unit 43 along with the compositeimage.

The object alignment unit 43 detects motion in the composite image andthe photographed image supplied from the motion blur elimination unit 42and, on the basis of a result of the motion detection, performsalignment between the composite image and the photographed image.

In other words, the object alignment unit 43 performs alignment to alignthe position of an object in the composite image with the same object inthe photographed image.

The object alignment unit 43 then supplies the aligned composite imageand photographed image to the composition unit 33.

The composition unit 33 generates a latest composite image bycompositing the composite image and the photographed image supplied fromthe alignment unit 32 (specifically the object alignment unit 43thereof).

Specifically, the composition unit 33 includes a feature datacalculation unit 51, a peak calculation unit 52 and an image compositionunit 53.

The feature data calculation unit 51 calculates feature data(hereinafter also referred to as in-focus feature data) representing adegree of focus (being in focus) for a pixel in each of the compositeimage and the photographed image supplied from the alignment unit 32,and supplies the in-focus feature data to the peak calculation unit 52.

The peak calculation unit 52 calculates a peak of the in-focus featuredata of the pixel arranged at the same position in each of the compositeimage and the photographed image. That is, the peak calculation unit 52detects the larger in-focus feature data between the in-focus featuredata of the pixel arranged at the same position in each of the compositeimage and the photographed image, and supplies a detected result(hereinafter also referred to as a detected peak) to the imagecomposition unit 53.

The image composition unit 53 generates the latest composite image bycompositing the composite image and the photographed image supplied fromthe alignment unit 32 in accordance with the detected peak supplied fromthe peak calculation unit 52.

The (latest) composite image obtained in the image composition unit 53is supplied to the frame buffer 31 and also output to the display device13 (FIG. 1) as appropriate.

The drive control unit 34 drives the optical system 22 to shift thefocus position.

The control unit 35 controls the entire signal processing device 12.

<Overview of Photographing Performed to Obtain Photographed Image andGenerating Composite Image>

FIG. 3 is a diagram illustrating an overview of photographing performedto obtain the photographed image in the photographing unit 11 andgenerating the composite image in the signal processing device 12.

FIG. 3 illustrates an example where the photographing unit 11 performsphotographing to obtain photographed image (frames) F1, F2, F3 and thelike at the frame rate of 120 Hz.

Moreover, as illustrated in FIG. 3, the focus position in obtaining thephotographed image is periodically shifted to four focus positions pos1,pos2, pos3 and pos4 in each frame.

Here, the focus positions pos1, pos2, pos3 and pos4 are different andhave relationship represented by expression pos1<pos2<pos3<pos4 in FIG.3.

The photographing unit 11 performs photographing to obtain thephotographed images F1, F2, F3 and the like while periodically changingthe focus position to the focus positions pos1, pos2, pos3, and pos4 asdescribed above.

As a result, the photographed image F1 is an image at the focus positionpos1 (image focused on the focus position pos1), the photographed imageF2 is an image at the focus position pos2, the photographed image F3 isan image at the focus position pos3, and a photographed image F4 is animage at the focus position pos4.

A photographed image F5 is an image focused on the focus position pos1and, in this manner, a photographed image from then on corresponds to animage focused on a focus position that changes periodically.

The signal processing device 12 theoretically generates a deep focuscomposite image by compositing a plurality of photographed images withdifferent focus positions such as four frames of photographed imagesfocused on the focus positions pos1, pos2, pos3, and pos4.

A first composition method and a second composition method can beadopted, for example, as an Extended Depth of Field (EDoF) method thatgenerates the deep focus composite image by compositing the plurality ofphotographed images with different focus positions such as thephotographed images focused on the focus positions pos1, pos2, pos3, andpos4.

In the first composition method, the photographed image focused on eachof the focus positions pos1 to pos3 is aligned with the photographedimage focused on the focus position pos4 that is the latest photographedimage among the photographed images focused on the focus positions pos1to pos4.

After that, among the pixels of the photographed images focused on thefocus positions pos1 to pos4 after alignment, a pixel of thephotographed image focused on the focus position with the maximum (peak)in-focus feature data is selected as a pixel of a composite image,whereby a composite image formed of such pixel is generated.

In the second composition method, for example, the photographed image F1at the focus position pos1 obtained first becomes the composite image asis. That composite image (last composite image) and the followingphotographed image F2 at the focus position pos2 are then composited.

In other words, the last composite image is aligned with thephotographed image F2 at the focus position pos2 being the latestphotographed image.

After that, between the pixels of the photographed image F2 at the focusposition pos2 and the composite image after alignment, a pixel of thephotographed image F2 or the composite image with the maximum (larger)in-focus feature data is selected as a pixel of a latest composite imageC1, whereby the latest composite image C1 formed of such pixel isgenerated.

As for the photographed image F3 at the focus position pos3 photographedafter the photographed image F2 at the focus position pos2, thecomposite image C1 serves as the last composite image C1 where thecomposite image C1 and the photographed image F3 (latest photographedimage) at the focus position pos3 are composited in the similar mannerto generate a latest composite image C2.

As for the photographed image F4 at the focus position pos4 photographedafter the photographed image F3 at the focus position pos3, thecomposite image C2 serves as the last composite image C2 where thecomposite image C2 and the photographed image F4 at the focus positionpos4 are composited in the similar manner to generate a latest compositeimage C3.

Likewise, as for a succeeding photographed image, a latest compositeimage is generated by compositing the last composite image and thelatest photographed image in the second composition method.

Each of the composite image obtained by compositing the photographedimages at the focus positions pos1 to pos4 in the first compositionmethod and the composite image C3 obtained by the second compositionmethod is a deep focus image with the depth of field including the focuspositions pos1 to pos4.

Moreover, in either the first or second composition method, thecomposite image can be obtained after the photographed images at thefocus positions pos1 to pos4 are obtained.

Note that in the first composition method, the four frames ofphotographed images at the focus positions pos1 to pos4 are subjected tothe composition processing (alignment processing performed in thealignment unit 32 and composition processing performed in thecomposition unit 33) that generates the composite image.

In the second composition method, on the other hand, the two frames ofimages including the last composite image and the latest photographedimage are subjected to the composition processing.

Therefore, while there is no difference in the depth of field of thecomposite image obtained in the first and second composition methods,the composition processing can be performed faster in the secondcomposition method than in the first composition method when three ormore frames of photographed images are used to generate the compositeimage.

According to the second composition method, as described above, thecomposition processing can be performed faster so that the compositeimage being a deep focus image can be obtained with a low delay and at ahigh frame rate.

Now, when an image is to be provided to a user such as a doctorperforming an operation with medical equipment, it is desirable from thenature of the medical field that the image is provided with a low delayand at a high frame rate as described above, in which case the secondcomposition method can be employed to be able to provide the deep focusimage with the low delay and at the high frame rate. As a result,observability of an object (ease of observing an object) as well as easeof performing an operation on the object can be increased.

Moreover, according to the second composition method, the compositionprocessing can be performed in a short time to be able to obtain thedeep focus image promptly even when focus drive in the photographingunit 11 is slow and takes time.

Note that when the photographing unit 11 performs photographing toobtain the photographed image at the frame rate of 120 Hz whileperiodically changing the focus position to the focus positions pos1,pos2, pos3, and pos4 as illustrated in FIG. 3, the signal processingdevice 12 can generate a single frame of a composite image bycompositing four frames of photographed images every time the fourframes of the photographed images at the focus positions pos1, pos2,pos3, and pos4 are obtained. In this case, there can be obtained acomposite image with the depth of field that is (approximately) fourtimes that of a photographed image at a single focus position and theframe rate of 30 Hz.

Moreover, when the photographing unit 11 performs photographing toobtain the photographed image at the frame rate of 120 Hz whileperiodically changing the focus position to the focus positions pos1,pos2, pos3, and pos4 as illustrated in FIG. 3, the signal processingdevice 12 can generate a single frame of a composite image bycompositing preceding four frames of the photographed images including alatest photographed image every time the latest photographed image isobtained. In this case, there can be obtained a composite image with thedepth of field that is (approximately) four times that of a photographedimage at a single focus position and the frame rate of 120 Hz.

While the medical observation system of FIG. 1 can adopt either thefirst or second composition method, there will be described an examplewhere the second composition method between the first and secondcomposition methods is adopted.

<Processing Performed by Alignment Unit 32 and Composition Unit 33>

FIG. 4 is a diagram illustrating an example of processing performed bythe alignment unit 32 and the composition unit 33 illustrated in FIG. 2.

The photographing unit 11 performs photographing to obtain thephotographed image while changing the focus position as described withreference to FIG. 3.

FIG. 4 illustrates a case where a photographed image including objectsobj1 and obj2 is obtained at times t0, t1, and t2.

Focus positions of the photographed images obtained at times t0 to t2are different so that, in the photographed image at time t0, the objectobj1 is in focus while the object obj2 is out of focus. In thephotographed images obtained at times t1 and t2, the object obj1 is outof focus while the object obj2 is in focus.

Here, there will be described the processing performed by the alignmentunit 32 and the composition unit 33 with reference to FIG. 4 assumingthat the photographed image at time t0 is the last composite image andthe photographed image at time t1 is the latest photographed image, inorder to simplify description.

The alignment unit 32 aligns the position of the photographed image attime t0 being the last composite image with the photographed image attime t1 being the latest photographed image.

The alignment is performed such that the identical objects in thephotographed images obtained at times t0 and t1 overlap each other asmuch as possible.

Specifically, the angle-of-view adjustment unit 41 of the alignment unit32 adjusts the angle of view of the photographed image at time t1 as acomposite image, for example, such that the objects obj1 as well as theobjects obj2 identical in the photographed images at times t0 and t1overlap each other as much as possible.

The angle of view is adjusted on the assumption that the photographedimage at time t1 being the latest photographed image is highlycorrelated with the photographed image at time t0 being the lastcomposite image, where the angle of view of the photographed image attime t0 is changed little by little to compute cross correlation or asum total of an absolute value of a difference in pixel values as acorrelation value representing correlation between the photographedimages at times t0 and t1.

Then there is computed the angle of view with which the correlationvalue between the photographed images at times t0 and t1 is the highest,and the angle of view of the photographed image at time t0 is adjustedto such angle of view (the photographed image at time t0 is scaled up ordown).

The photographed image being obtained while changing the focus position,the angle of view may vary slightly between one photographed image and anext photographed image due to the shift in the focus position. Theangle of view is adjusted in order to correct such variation in theangle of view.

After adjusting the angle of view, the motion blur elimination unit 42of the alignment unit 32 eliminates motion blur in the photographedimage at time t1 being the latest photographed image.

An arbitrary method can be adopted to eliminate the motion blur. When ablur kernel can possibly be the filter causing the motion blur, forexample, the motion blue can be eliminated by deconvolution of the blurkernel.

Note that the processing of eliminating the motion blur in the motionblur elimination unit 42 can be skipped. That is, the processing ofeliminating the motion blur can be skipped when the motion blur iseliminated by deconvolution of the blur kernel which, however, is not beassumed as the filter causing the blur, for example.

While the motion blur is eliminated only from the latest photographedimage in this case, the motion blur can also be eliminated from the lastcomposite image. However, the motion blur is already eliminated from thelast composite image since it is obtained by compositing the deblurredphotographed image and a composite image obtained before the lastcomposite image except for a case where the photographed image serves asthe last composite image as is. Therefore, the last composite image doesnot have to be subjected to elimination of the motion blur except forthe case where the photographed image serves as the last composite imageas is.

The object alignment unit 43 of the alignment unit 32 thereafter alignsthe position of an object in the photographed image at time t0 being thelast composite image with the position of an object in the photographedimage at time t1 being the latest photographed image.

The object alignment unit 43 performs alignment on the assumption thatthe photographed image at time t1 being the latest photographed imageand the photographed image at time t0 being the last composite image donot vary much (do not have a big difference) in the depth of field,namely, the identical object is in focus in both the photographed imageat time t1 being the latest photographed image and the photographedimage at time t0 being the last composite image.

The object alignment unit 43 performs alignment by detecting motionbetween the photographed image at time t1 being the latest photographedimage and the photographed image at time t0 being the last compositeimage pixel by pixel, for example.

The motion detection can be performed by an arbitrary method, forexample. The motion detection can be performed by block matching or aKanade Lucas Tomasi (KLT) method based on a feature point, for example.

In the alignment performed by the object alignment unit 43, a motionvector is detected pixel by pixel in the motion detection, and then themotion vector detected pixel by pixel is used to find one or a pluralityof representative vectors representing motion from one or a plurality ofpoints in the photographed image at time t0 being the last compositeimage to one or a plurality of points in the photographed image at timet1 being the latest photographed image.

Then, a projection transformation matrix that realizes projectiontransformation matching the motion represented by the representativevector is computed, so that the photographed image at time t0 being thelast composite image is subjected to projection transformation accordingto the projection transformation matrix to align the position of thephotographed image at time t0 being the last composite image with thephotographed image at time t1 being the latest photographed image.

The photographed image at time t0 being the last composite image and thephotographed image at time t1 being the latest photographed image afterthe alignment are then supplied from the alignment unit 32 to thecomposition unit 33 and composited.

That is, the feature data calculation unit 51 of the composition unit 33calculates in-focus feature data representing the degree to which apixel is in focus in each of the photographed image at time t1 being thelatest photographed image and the photographed image at time t0 beingthe last composite image, and supplies a feature data image having thecalculated in-focus feature data as a pixel value to the peakcalculation unit 52.

The in-focus feature data can be feature data having a large value foran in-focus pixel and a small value for a blurred pixel, for example.Laplacian can be adopted as such in-focus feature data, for example.

The peak calculation unit 52 refers to the feature data image from thefeature data calculation unit 51 and calculates (detects) a peak of thein-focus feature data in the pixel at the identical position in each ofthe photographed image at time t1 being the latest photographed imageand the photographed image at time t0 being the last composite image.

FIG. 4 illustrates a pixel p1 corresponding to the object obj1 in thephotographed image at time t1 being the latest photographed image and apixel p0 corresponding to the object obj1 in the photographed image attime t0 being the last composite image and located in the positionidentical to the pixel p1, where the in-focus feature data of the pixelp0 corresponding to the object obj1 that is in focus is larger than thefeature data of the pixel p1 corresponding to the object obj1 that isout of focus. As a result, the in-focus feature data of the pixel p0 isdetected as the peak of the in-focus feature data of the pixels p0 andp1 at the identical position, and the detected peak being a result ofthe detection is supplied from the peak calculation unit 52 to the imagecomposition unit 53.

The image composition unit 53 generates the latest composite image bycompositing the photographed image at time t1 being the latestphotographed image and the photographed image at time t0 being the lastcomposite image according to the detected peak from the peak calculationunit 52.

Specifically, the image composition unit 53 generates the latestcomposite image by selecting, as a pixel of the latest composite image,a pixel with the larger in-focus feature data being the detected peak,namely a pixel corresponding to the object that is more in-focus,between the pixels at the identical position in each of the photographedimage at time t1 being the latest photographed image and thephotographed image at time t0 being the last composite image.

The composition unit 33 as described above composites the latestphotographed image and the last composite image that are aligned by themotion detection in the alignment unit 32. The latest deep-focuscomposite image can thus be generated by following motion even when anobject with some degree of motion is in the latest photographed imageand the last composite image.

FIG. 5 is a diagram illustrating a photographed image that is obtainedwhile changing a focus position in the photographing unit 11.

FIG. 5 illustrates a case where objects obj1, obj2, and obj3 arranged ina real space are photographed while changing the focus position.

Note that the objects obj1, obj2, and obj3 are arranged in this orderaway from the side of the photographing unit 11.

The focus position is shifted from a front side to a back side (as seenfrom the side of the photographing unit 11) in photographing the objectsobj1 to obj3.

Photographed images F#N and F#N+1 in FIG. 5 are adjacent frames ofphotographed images, where the photographed image F#N+1 is obtainedafter the photographed image F#N.

The depth of field when the photographed image F#N is obtained coversthe forefront object obj1 and the second forefront object obj2 but doesnot cover the object obj3 arranged farthest back. Therefore, the objectsobj1 and obj2 are in focus while the object obj3 is out of focus in thephotographed image F#N.

On the other hand, the depth of field when the photographed image F#N+1is obtained covers the second forefront object obj2 and the object obj3arranged farthest back but does not cover the forefront object obj1.Therefore, the objects obj2 and obj3 are in focus while the object obj1is out of focus in the photographed image F#N+1.

The object obj2 is thus in focus in both of the photographed images F#Nand F#N+1.

The focus position is shifted such that, as described above, one or moreobjects (the object obj2 in FIG. 5) is/are in focus in the adjacentframes of the photographed images F#N and F#N+1, namely, the depths offield of the adjacent frames of the photographed images F#N and F#N+1overlap in part.

<First Operational Example of Medical Observation System>

FIG. 6 is a flowchart illustrating a first operational example of themedical observation system illustrated in FIG. 1.

Specifically, FIG. 6 illustrates an operational example of the medicalobservation system when the signal processing device 12 is configured asillustrated in FIG. 2.

In step S11, the control unit 35 sets a target value of the focusposition to a default value such as a minimum value of a range withinwhich the focus position can be shifted, then the operation proceeds toprocessing in step S12.

In step S12, the control unit 35 controls the drive control unit 34 toshift the focus position to the target value, then the operationproceeds to processing in step S13.

In step S13, the photographing unit 11 performs photographing to obtaina photographed image while the focus position is at the target value andsupplies the photographed image to the frame buffer 31, then theoperation proceeds to processing in step S14.

In step S14, the frame buffer 31 stores the photographed image from thephotographing unit 11 as an image of interest, then the operationproceeds to processing in step S15.

In step S15, the alignment unit 32 performs alignment between the imageof interest being the latest photographed image stored in the framebuffer 31 and the last composite image stored in the frame buffer 31 asdescribed with reference to FIG. 4. Moreover, in step S15, the alignmentunit 32 supplies the aligned image of interest and last composite imageto the composition unit 33, then the operation proceeds to processing instep S16.

Here, the composite image stored in the frame buffer 31 is reset, namelydeleted from the frame buffer 31, at a predetermined timing.

The composite image is reset at the start of photographing, for example.

Therefore, at the start of photographing, the composite image is resetand not stored in the frame buffer 31.

The composite image can also be reset in step S11 where the target valueof the focus position is set to the default value, for example.Moreover, the composite image can be reset when a photographed image notsuitable for generating a deep focus composite image is obtained such aswhen large motion is detected from the photographed image, and acomposition restriction condition that restricts composition of thephotographed image (and resultant generation of a composite image) issatisfied.

The processing in each of steps S15 and S16 is skipped when thecomposite image is not stored in the frame buffer 31, in which case theimage of interest is stored as the composite image into the frame buffer31.

In step S16, as described above with reference to FIG. 4, thecomposition unit 33 calculates the in-focus feature data of the pixel ineach of the aligned image of interest and last composite image and,according to the calculated in-focus feature data, composites the imageof interest and the last composite image to generate the latestcomposite image.

Specifically, the composition unit 33 generates the latest compositeimage by selecting, as a pixel of the latest composite image, the pixelthat is more in focus between the pixels in the image of interest andthe last composite image according to the in-focus feature data.

The composition unit 33 supplies the latest composite image to the framebuffer 31, which stores the latest composite image by overwriting thelast composite image therewith, then the operation proceeds from theprocessing in step S16 to processing in step S17.

Here, the latest composite image stored in the frame buffer 31 asdescribed above is used as a last composite image in step S15 performedin the next round of operation.

In step S17, the control unit 35 determines whether the target value ofthe focus position is set to a maximum value of the range within whichthe focus position can be shifted.

The operation proceeds to processing in step S18 when it is determinedin step S17 that the target value is not set to the maximum value of therange within which the focus position can be shifted, namely when thetarget value is smaller than the maximum value of the range within whichthe focus position can be shifted.

In step S18, the control unit 35 increases the target value of the focusposition by a predetermined value from the current value, then theoperation returns to the processing in step S12.

In step S12, as described above, the control unit 35 controls the drivecontrol unit 34 to shift the focus position to the target value. Thesimilar processing is repeated from then on so that the photographedimage is obtained while changing the focus position and that the latestphotographed image and the last composite image are composited.

On the other hand, the operation proceeds to processing in step S19 whenit is determined in step S17 that the target value is set to the maximumvalue of the range within which the focus position can be shifted,namely when a plurality of photographed images is obtained whileshifting the focus position across the range within which the focusposition can be shifted.

In step S19, the composition unit 33 outputs the latest composite imageto be displayed in the display device 13 (FIG. 1), then the operationreturns to the processing in step S11.

In the first operational example, the plurality of photographed imagesis obtained while shifting the focus position across the range withinwhich the focus position can be shifted, and then the composite image isgenerated by using all of the plurality of photographed images andoutput/displayed to/in the display device 13.

Therefore, in the first operational example, the frame rate of thecomposite image displayed in the display device 13 is lower than theframe rate of the photographed image obtained in the photographing unit11 by the amount corresponding to the number of frames of thephotographed images used to generate the composite image.

<Second Operational Example of Medical Observation System>

FIG. 7 is a flowchart illustrating a second operational example of themedical observation system illustrated in FIG. 1.

Specifically, FIG. 7 illustrates another operational example of themedical observation system when the signal processing device 12 isconfigured as illustrated in FIG. 2.

In the second operational example, processing similar to the processingperformed in each of steps S11 to S16 of the first operational examplein FIG. 6 is performed in each of steps S21 to S26.

In step S26, the composition unit 33 generates a latest composite imageand supplies it to the frame buffer 31, which stores the latestcomposite image by overwriting the last composite image therewith, thenthe operation proceeds to processing in step S27.

In step S27, the composition unit 33 outputs the latest composite imageto be displayed in the display device 13 (FIG. 1) as with step S19 ofthe first operational example in FIG. 6, then the operation proceeds toprocessing in step S28.

In each of steps S28 and S29, processing similar to the processingperformed in each of steps S17 and S18 of the first operational examplein FIG. 6 is performed.

In the first operational example of FIG. 6, as described above, theplurality of photographed images is obtained while shifting the focusposition across the range within which the focus position can beshifted, and then the composite image is generated by using all of theplurality of photographed images and output/displayed to/in the displaydevice 13.

On the other hand, what is common to the first operational example inthe second operational example of FIG. 7 is that a plurality ofphotographed images is obtained while shifting the focus position acrossthe range within which the focus position can be shifted.

However, in the second operational example, a latest composite image isoutput/displayed to/in the display device 13 in step S27 every time thelatest composite image is generated by using the latest photographedimage (image of interest) in step S26 corresponding to S16 of the firstoperational example.

Therefore, in the second operational example, the frame rate of thecomposite image displayed in the display device 13 corresponds with theframe rate of the photographed image obtained by the photographing unit11.

Note that a user can perform an operation or the like to select whetherto generate the composite image by using all of the plurality ofphotographed images obtained while shifting the focus position acrossthe range within which it can be shifted and output the composite imageto the display device 13 as described in the first operational exampleof FIG. 6, or to output the latest composite image to the display device13 every time the latest composite image is generated by using thelatest photographed image as described in the second operational exampleof FIG. 7.

Between the first and second operational examples, there will bedescribed an example where, as described in the first operationalexample, the composite image is generated by using all of the pluralityof photographed images obtained while shifting the focus position acrossthe range within which the focus position can be shifted and then outputto the display device 13.

<Third Operational Example of Medical Observation System>

FIG. 8 is a flowchart illustrating a third operational example of themedical observation system illustrated in FIG. 1.

Specifically, FIG. 8 illustrates yet another operational example of themedical observation system when the signal processing device 12 isconfigured as illustrated in FIG. 2.

In the third operational example, processing similar to the processingperformed in each of steps S11 to S18 of the first operational examplein FIG. 6 is performed in each of steps S31 to S38.

In step S37, as with the corresponding step S17 of the first operationalexample, the control unit 35 determines whether the target value of thefocus position is set to a maximum value of the range within which thefocus position can be shifted.

The operation proceeds to processing in step S39 when it is determinedin step S37 that the target value is set to the maximum value of therange within which the focus position can be shifted.

In step S39, the control unit 35 determines whether a compositionrestriction condition that restricts composition of the photographedimage (and resultant generation of a composite image) is satisfied.

Here, the composition restriction condition can be a case where aphotographed image not suitable for generating a deep focus compositeimage is obtained or a case where a user does not desire to obtain adeep focus image, for example.

The photographed image not suitable for generating the deep focuscomposite image is obtained when, for example, reliability of the angleof view adjustment performed on the image of interest and the lastcomposite image is less than or equal to a threshold, the angle of viewadjustment being performed in the alignment between the image ofinterest and the last composite image in step S35.

In the angle of view adjustment, as described with reference to FIG. 4,there is obtained the angle of view with the highest correlation valuebetween the image of interest (photographed image at time t1 being thelatest photographed image) and the last composite image (photographedimage at time t0 being the last composite image), and the angle of viewof the last composite image is adjusted to the angle of view obtained.

The reliability of the angle of view adjustment can be indicated by thecorrelation value between the image of interest and the last compositeimage when the angle of view of the last composite image is adjusted,for example.

Moreover, the photographed image not suitable for generating the deepfocus composite image is obtained when, for example, reliability of themotion detection performed on the image of interest and the lastcomposite image is less than or equal to a threshold, the motiondetection being performed in compositing the image of interest and thelast composite image in step S36.

The reliability of the motion detection can be indicated by a valueinversely proportional to a Sum of Absolute Difference (SAD) or the likebeing an evaluation value that is used to detect a motion vector inblock matching performed as motion detection and evaluates similaritybetween blocks, for example.

Furthermore, the photographed image not suitable for generating the deepfocus composite image is obtained when, for example, the degree of themotion detected in the image of interest and the last composite image inthe motion detection is higher than or equal to a threshold, the motiondetection being performed in compositing the image of interest and thelast composite image in step S36.

When a user systematically performs panning or zooming by operating thephotographing unit 11 or when the degree of motion of an object is largerelative to shift speed of the focus position, for example, it is moredifficult to align the objects in the image of interest and the lastcomposite image, thereby possibly causing considerable motion blur in acomposite image.

The case where the degree of motion detected in the image of interestand the last composite image in the motion detection is higher than orequal to the threshold can be set as the composition restrictioncondition to be able to prevent generation of the aforementionedcomposite image with considerable motion blur.

The user does not desire to obtain the deep focus image when, forexample, a photographed image includes a living body such as a surgicalsite of a human body undergoing an operation as well as a treatment toolsuch as forceps used to perform a treatment on the surgical site wherethe treatment tool is intentionally moved (the treatment tool is movedtoward the surgical site, for example) by the user (who operates thetreatment tool).

One can see whether the treatment tool is in motion by image recognitionrecognizing that the treatment tool is included in the photographedimage and detecting motion of the treatment tool, for example.

One can also see whether the treatment tool is in motion on the basis ofa state of a button (not shown) operated to be in an on state when thetreatment tool is handled by the user, for example.

Moreover, the user does not desire to obtain the deep focus image when,for example, a button (not shown) operated when the user does not desireto obtain the deep focus image is operated.

The operation proceeds to processing in step S40 when it is determinedin step S39 that the composition restriction condition is satisfied.

In step S40, the composition unit 33 reads from the frame buffer 31 oneof the plurality of photographed images used to generate the latestcomposite image, namely a single frame of photographed image focused onthe center or the like through the alignment unit 32.

The composition unit 33 then selects the photographed image read fromthe frame buffer 31 and focused on the center as the latest compositeimage, then the operation proceeds from the processing in step S40 toprocessing in step S41.

On the other hand, the operation skips the processing in step S40 andproceeds to the processing in step S41 when it is determined in step S39that the composition restriction condition is not satisfied.

In step S41, the composition unit 33 outputs the latest composite imageto be displayed in the display device 13 (FIG. 1) as with step S19 ofthe first operational example in FIG. 6, then the operation returns tothe processing in step S31.

In the third operational example of FIG. 8, the composite image formedby compositing the plurality of photographed images obtained whilechanging the focus position is output/displayed to/in the display device13 when the composition restriction condition is not satisfied whereas,when the composition restriction condition is satisfied, one of theplurality of photographed images is output/displayed to/in the displaydevice 13 due to output restriction on the composite image formed bycompositing the plurality of photographed images obtained while changingthe focus position.

As a result, there can be prevented a case where the composite imageformed by compositing the plurality of photographed images obtainedwhile changing the focus position is displayed in the display device 13when the user intentionally moves the treatment tool largely toward thesurgical site and does not particularly feel the need of EDoF, forexample.

Moreover, there can be prevented a case where an image with considerablemotion blur is displayed in the display device 13 when the plurality ofphotographed images with a large degree of motion caused by strongshaking of the photographing unit 11 is obtained and composited, forexample.

On the other hand, it is desirable for the medical observation systembeing medical equipment to prevent interruption of the image displayedin the display device 13 as much as possible and keep displaying theimage in the display device 13 considering the nature of the system.

In the third operational example of FIG. 8, the photographed imagefocused on the center instead of the composite image with considerablemotion blur is displayed in the display device 13 when the compositionrestriction condition is satisfied, namely when the composite image withthe considerable motion blur caused by strong shaking of thephotographing unit 11 is generated, for example.

As a result, there can be prevented displaying of the composite imagewith considerable motion blur in the display device 13 as well asinterruption of the image displayed in the display device 13.

<Second Configuration Example of Signal Processing Device 12>

FIG. 9 is a block diagram illustrating a second configuration example ofthe signal processing device 12 in FIG. 1.

Note that in the figure, a part corresponding to the one in FIG. 2 isassigned the same reference numeral as that in FIG. 2 to omitdescription of such part as appropriate.

The signal processing device 12 in FIG. 9 includes a frame buffer 31 toa control unit 35 as well as a depth estimation unit 61, a range settingunit 62 and a range storing unit 63.

Therefore, what is common to FIG. 2 in FIG. 9 is that the signalprocessing device 12 includes the frame buffer 31 to the control unit35.

The signal processing device 12 in FIG. 9 is however different from thatin FIG. 2 in that the depth estimation unit 61, the range setting unit62 and the range storing unit 63 are newly provided.

The depth estimation unit 61 estimates the depth of an object in aphotographed image obtained by a photographing unit 11 and supplies adepth map in which depth information indicating the depth is registeredto the range setting unit 62.

Here, for example, the depth of the object can be estimated from aparallax between an L image and an R image forming a 3D imagephotographed by the photographing unit 11 when the photographing unit 11is a so-called 3D camera capable of photographing a 3D image.

The depth can also be estimated by measuring Time of Flight (ToF) withuse of a laser or irradiating the object with a specific pattern such astextured light, for example. Moreover, the depth of the object can beestimated on the basis of a state of an optical system 22 controlled byan Auto Focus (AF) function when the medical observation system of FIG.1 is equipped with the AF function.

The range setting unit 62 uses the depth map from the depth estimationunit 61 as appropriate, sets a range within which the focus position isshifted (hereinafter also referred to as a focus shift range) accordingto an operation of a user or the like, and supplies the range to therange storing unit 63.

The range storing unit 63 stores the focus shift range supplied from therange setting unit 62.

While the drive control unit 34 of the signal processing device 12 inFIG. 2 shifts the focus position across the range within which the focusposition can be shifted (from the minimum value to the maximum value ofthe range within which the focus position can be shifted), a drivecontrol unit 34 of the signal processing device 12 in FIG. 9 shifts thefocus position across the focus shift range stored in the range storingunit 63.

Accordingly, the photographing unit 11 of FIG. 9 performs photographingto obtain a photographed image while changing the focus position withinthe focus shift range set according to the user operation.

FIG. 10 is a diagram illustrating an example of setting the focus shiftrange in the range setting unit 62.

As with FIG. 5, objects obj1, obj2, and obj3 in FIG. 10 are arranged inthis order toward the back in a real space.

Then, as illustrated in FIG. 10, the focus shift range is set such thatpositions of the two objects obj2 and obj3 at the back among the objectsobj1 to obj3 are included as the focus positions.

Assuming that the positions of the objects obj1 to obj3 are included inthe range within which the focus position can be shifted and when thefocus shift range is set to the range within which the focus positioncan be shifted, an image in which all the objects obj1 to obj3 are infocus is generated as a composite image.

On the other hand, when the focus shift range is set to include thepositions of the two objects obj2 and obj3 among the objects obj1 toobj3 as illustrated in FIG. 10, an image in which the two objects obj2and obj3 out of the objects obj1 to obj3 are in focus is generated as acomposite image.

As a result, the focus position in obtaining the photographed image usedto generate the composite image can be limited by setting the focusshift range according to the user operation as described above. Thelimitation on the focus position can reduce the number of frames of thephotographed images used to generate the composite image and, as aresult, a composite image to be displayed in a display device 13 can begenerated at a shorter interval to be able to increase the frame rate ofthe composite image to a high frame rate.

<Fourth Operational Example of Medical Observation System>

FIG. 11 is a flowchart illustrating a fourth operational example of themedical observation system illustrated in FIG. 1.

Specifically, FIG. 11 illustrates an operational example of the medicalobservation system when the signal processing device 12 is configured asillustrated in FIG. 9.

In the fourth operational example, the depth estimation unit 61 in stepS51 estimates the depth, generates a depth map in which depthinformation of an object is registered and supplies the depth map to therange setting unit 62, then the operation proceeds to processing in stepS52.

In step S52, the range setting unit 62 waits for a user operation or thelike and, according to the operation, sets a focus shift range withinwhich the focus position is shifted and supplies it to the range storingunit 63, then the operation proceeds to processing in step S53.

Here, the user can specify the focus shift range by operating a touchpanel that is not shown or the like.

The user can specify the focus shift range by inputting an absolutedistance in millimeters (mm) as the minimum value and maximum value ofthe focus shift range, for example. The user can also specify the focusshift range by inputting the range toward the front and back in thedepth direction from the center being the focus position (in-focusposition) determined by AF, for example.

The user can also specify the focus shift range by specifying the objectin the photographed image obtained by the photographing unit 11, forexample.

In this case, the range setting unit 62 uses the depth map obtained inthe depth estimation unit 61 and sets the focus shift range.

That is, when the user specifies the object in an image displayed in thedisplay device 13, for example, the range setting unit 62 refers to thedepth map and acknowledges a range in the depth direction where theobject specified by the user is present. The range setting unit 62 thensets the range in the depth direction where the object specified by theuser is present as the focus shift range.

When the user specifies a plurality of objects, the range setting unit62 sets, as the focus shift range, a range between positionscorresponding to the forefront object and the object located farthestback among the plurality of objects.

Note that the focus shift range is set within the range the focusposition can be shifted.

The signal processing device 12 of FIG. 9 can also be configured withoutincluding the depth estimation unit 61 when the range setting unit 62does not use the depth map in setting the focus shift range.

In step S53, the range storing unit 63 stores the focus shift rangesupplied from the range setting unit 62.

Here, the focus shift range stored in the range storing unit 63 isupdated every time the user performs an operation to specify the focusshift range.

Following step S53, the operation proceeds to processing in step S61,from which on the focus position is shifted across the focus shift rangestored in the range storing unit 63 to obtain a photographed image andgenerate a composite image.

Specifically, a control unit 35 in step S61 sets a target value of thefocus position to a default value such as the minimum value of the focusshift range stored in the range storing unit 63, then the operationproceeds to processing in step S62.

In each of steps S62 to S66, processing similar to the processingperformed in each of steps S12 to S16 of the first operational examplein FIG. 6 is performed.

The operation then proceeds from the processing in step S66 toprocessing in step

S67, in which the control unit 35 determines whether the target value ofthe focus position is set to the maximum value of the focus shift rangestored in the range storing unit 63.

The operation proceeds to processing in step S68 when it is determinedin step S67 that the target value is not set to the maximum value of thefocus shift range, namely when the target value is smaller than themaximum value of the focus shift range.

As with step S18 of the first operational example in FIG. 6, the controlunit 35 in step S68 increases the target value of the focus position bya predetermined value from the current value, then the operation returnsto the processing in step S62, from which on the similar processing isrepeated.

Accordingly, the photographed image is obtained while changing the focusposition across the focus shift range set according to the useroperation, and then the composite image is generated.

On the other hand, the operation proceeds to processing in step S69 whenit is determined in step S67 that the target value is set to the maximumvalue of the focus shift range, namely when a plurality of photographedimages is obtained while shifting the focus position across the focusshift range.

In step S69, a composition unit 33 outputs a latest composite image tobe displayed in the display device 13 (FIG. 1) as with step S19 of thefirst operational example in FIG. 6, then the operation returns to theprocessing in step S61.

In the fourth operational example, as described above, the plurality ofphotographed images is obtained while shifting the focus position acrossthe focus shift range set according to the user operation, and thecomposite image is generated by using the plurality of photographedimages.

As a result, there can be displayed a composite image in which only theobject in the depth range intended (desired) by the user is in focus.

Moreover, the time it takes to shift the focus position is reduced whenthe focus shift range set according to the user operation is narrowerthan the range within which the focus position can be shifted, wherebythe frame rate of the composite image displayed in the display device 13can be increased to a high frame rate.

<Third Configuration Example of Signal Processing Device 12>

FIG. 12 is a block diagram illustrating a third configuration example ofthe signal processing device 12 of FIG. 1.

Note that in the figure, a part corresponding to the one in FIG. 2 isassigned the same reference numeral as that in FIG. 2 to omitdescription of such part as appropriate.

The signal processing device 12 in FIG. 12 includes a frame buffer 31 toa control unit 35 as well as a score calculation unit 71, an AF controlunit 72, a buffer 73 and a peak detection unit 74.

Therefore, what is common to FIG. 2 in FIG. 12 is that the signalprocessing device 12 includes the frame buffer 31 to the control unit35.

The signal processing device 12 in FIG. 12 is however different fromthat in FIG. 2 in that the score calculation unit 71, the AF controlunit 72, the buffer 73 and the peak detection unit 74 are newlyprovided.

The signal processing device 12 in FIG. 12 has an AF function.

Specifically, the score calculation unit 71 calculates a focus scorethat evaluates focus in a (latest) photographed image stored in theframe buffer 31.

The focus score can be indicated by a physical quantity representing thedegree of contrast in the photographed image, for example. A contrast AFmethod is employed in this case.

The score calculation unit 71 sets an AF frame demarcating the range ofthe photographed image for which the focus score is calculated at apredetermined position, or at the center of the photographed image, forexample. The score calculation unit 71 then uses the photographed imagewithin the AF frame to calculate the focus score and supplies the scoreto the AF control unit 72 and the buffer 73.

The AF control unit 72 controls the AF according to the focus scoresupplied from the score calculation unit 71.

Specifically, the AF control unit 72 determines a shift amount(including a direction) of the focus position such that the focusposition is shifted to have a higher focus score, and controls the drivecontrol unit 34 such that the focus position is shifted by the shiftamount.

The buffer 73 stores the focus score from the score calculation unit 71.The buffer 73 can be formed of a First In First Out (FIFO) memory with2N+1 tiers, for example, in which case the buffer 73 can store latest2N+1 frames of focus scores.

The peak detection unit 74 detects a peak, namely a local maximum scorebeing a local maximum value (including the maximum value), from thefocus score stored in the buffer 73 and supplies the detected localmaximum score to the control unit 35.

FIG. 13 is a graph illustrating an example of a relationship between thefocus position and the focus score.

FIG. 13 illustrates an example where the focus position is shifted inthe order of positions P1, P2, P3, P4, P5, and P6 by the contrast AFmethod and shifted to the in-focus position P6 in the end at which thefocus score has the maximum value.

That is, in the contrast AF method, the focus position is shifted tohave a higher focus score until the focus position is shifted to thevicinity of the in-focus position P6. Once the focus position reachesthe vicinity of the in-focus position P6, the focus position is shiftedto straddle the in-focus position P6 (to go back and forth over thein-focus position P6) in order to detect the in-focus position P6.

FIG. 13 illustrates the case where the focus position is first shiftedto the right in the figure in the order of the positions P1, P2, and P3.The focus score increasing as the focus position is shifted from theposition P1 to the position P2 decreases at the position P3, whereby thefocus position is shifted to the left in a reverse direction from theposition P3 to the position P4. After that, the focus position is againshifted to the right from the position P4 to the position P5, and againto the left from the position P5 to reach the in-focus position P6.

Therefore, in the contrast AF method as described above, it takes timefor the focus position to be shifted to the in-focus position P6 sincethe focus position is shifted to straddle the in-focus position P6 inthe vicinity thereof.

The signal processing device 12 in FIG. 12 detects the peak of the focusscore, namely the local maximum score (not necessarily the maximumvalue), from the focus score and generates a composite image by usingcomposition target images being a plurality of photographed imagesobtained at the focus position that is the focus position correspondingto the local maximum score and is in a predetermined range including apeak position.

Here, the focus score increasing from the position P1 to the position P2decreases at the position P3 in FIG. 13, whereby it is detected that thefocus score at the position P2 is the local maximum score and thus theposition P2 is the peak position.

Once the local maximum score is detected, the signal processing device12 in FIG. 12 stops shifting the focus position performed as AF.Moreover, a predetermined range R including the peak position P2 that isthe position P2 at which the local maximum score is detected is set as acomposition target focus range R being the range of the focus positionof the photographed images to be the composition target images.

The composite image is then generated by using the composition targetimages being the photographed images obtained at the focus position inthe composition target focus range R.

Note that the predetermined range R is set within the range in which thefocus position can be shifted.

When the composite image is generated as described above in conjunctionwith the AF function and by using the composition target images beingthe plurality of photographed images obtained at the focus position inthe predetermined range R that includes the peak position of the focusscore, a deep focus composite image can be obtained by using thephotographed images obtained up until the focus position is shifted tothe in-focus position P6 by the AF function.

Therefore, the focus position need only be shifted to the vicinity ofthe in-focus position P6, not to the in-focus position P6, in AF so thatAF can be substantially increased in speed.

Moreover, there can be prevented a case where, in generating a compositeimage, a photographed image is obtained at a focus position away fromthe position of the object in a real space photographed by thephotographing unit 11. In other words, there can be prevented a casewhere a photographed image not focused on any object is obtained. Thecomposite image can be generated faster as a result, and thus the framerate of the composite image can be increased to a high frame rate.

<Fifth Operational Example of Medical Observation System>

FIG. 14 is a flowchart illustrating a fifth operational example of themedical observation system illustrated in FIG. 1.

Specifically, FIG. 14 illustrates an operational example of the medicalobservation system when the signal processing device 12 is configured asillustrated in FIG. 12.

In step S71, a photographing unit 11 performs photographing to obtain aphotographed image and supplies the photographed image to the framebuffer 31, then the operation proceeds to processing in step S72.

In step S72, the frame buffer 31 stores the photographed image suppliedfrom the photographing unit 11, then the operation proceeds toprocessing in step S73.

In step S73, the score calculation unit 71 calculates a focus score byusing a photographed image within the AF frame set at a predeterminedposition among the (latest) photographed images stored in the framebuffer 31 and supplies the focus score to the AF control unit 72 and thebuffer 73, then the operation proceeds to processing in step S74.

In step S74, the buffer 73 stores the focus score supplied from thescore calculation unit 71, then the operation proceeds to processing instep S75.

In step S75, the peak detection unit 74 performs detection of a localmaximum score from the focus score stored in the buffer 73 anddetermines whether the local maximum score is successfully detected.

The operation proceeds to processing in step S76 when it is determinedin step S75 that the local maximum score is not successfully detected.In step S76, the composition unit 33 reads the latest photographed imagefrom the frame buffer 31 through the alignment unit 32 and outputs thelatest photographed image to a display device 13 as a latest compositeimage, then the operation proceeds to processing in step S77. Step S76can be skipped here.

In step S77, the AF control unit 72 determines a shift amount of thefocus position to have a higher focus score according to the focus scoresupplied from the score calculation unit 71, then the operation proceedsto processing in step S78.

In step S78, the AF control unit 72 controls the drive control unit 34to shift the focus position by the shift amount determined in step S77,whereby the focus position is shifted by the shift amount determined instep S77.

The operation thereafter returns from the processing in step S78 to stepS71, and the processing in each of steps S71 to S78 is repeated until itis determined in step S75 that the local maximum score is detected.

The operation then proceeds to processing in step S80 when it isdetermined in step S75 that the local maximum score is detected.

In step S80, the control unit 35 sets a predetermined range R of thefocus position with the center being a peak position that is the focusposition at which the local maximum score is detected as a compositiontarget focus range R being the range of the focus position of thephotographed images to be used as composition target images.

Moreover, from among the photographed images obtained at the focuspositions within the composition target focus range R, the control unit35 identifies 2N+1 frames of photographed images as the compositiontarget images, the 2N+1 frames of photographed images including 2Nframes of photographed images obtained at N focus positions precedingand following the peak position and a single frame of photographed imageobtained at the focus position being the peak position.

Note that when there is less than N frames of photographed imagesobtained at the focus positions preceding or following the peak positionamong the photographed images obtained at the focus positions within thecomposition target focus range R, all photographed images obtained atthe focus positions preceding or following the peak position can beidentified as the composition target images among the photographedimages obtained at the focus positions within the composition targetfocus range R, for example.

Alternatively, the photographed images can be obtained while shiftingthe focus position within the composition target focus range R such thatN frames of photographed images are obtained at the focus positionspreceding and following the peak position.

The operation proceeds to processing in step S81 after the compositiontarget images are identified as described above in step S80.

Here, in the fifth operational example of FIG. 14, the composite imagestored in the frame buffer 31 can be reset when the composition targetimages are identified in step S80.

In step S81, the alignment unit 32 selects, as an image of interest, onephotographed image not yet selected as the image of interest from amongthe photographed images being the composition target images stored inthe frame buffer 31, then the operation proceeds to processing in stepS82.

As with step S15 of the first operational example in FIG. 6, thealignment unit 32 in step S82 performs alignment between the image ofinterest and the last composite image stored in the frame buffer 31 andsupplies the aligned image of interest and last composite image to thecomposition unit 33, then the operation proceeds to processing in stepS83.

Here, as is the case with the example in FIG. 6, the processing in eachof steps S82 and S83 is skipped when the composite image is not storedin the frame buffer 31, in which case the image of interest is stored asthe composite image into the frame buffer 31.

As with step S16 of the first operational example in FIG. 6, thecomposition unit 33 in step S83 calculates in-focus feature data of apixel in each of the aligned image of interest and last composite imageand, according to the in-focus feature data, composites the image ofinterest and the last composite image to generate a latest compositeimage.

The composition unit 33 also supplies the latest composite image to theframe buffer 31, which stores the latest composite image by overwritingthe last composite image therewith, then the operation proceeds from theprocessing in step S83 to processing in step S84.

In step S84, the alignment unit 32 determines whether all thecomposition target images are selected as the images of interest.

When it is determined in step S84 that not all the composition targetimages are selected as the images of interest yet, the operation returnsto the processing in step S81, from which on the similar processing isrepeated.

On the other hand, the operation proceeds to processing in step S85 whenit is determined in step S84 that all the composition target images areselected as the images of interest, or when a composite image using allthe composition target images is generated as the latest compositeimage.

In step S85, the composition unit 33 outputs the latest composite imageto be displayed in the display device 13, then the operation returns tothe processing in step S71

Note that while the focus position is shifted until the local maximumscore, namely the peak position, is detected as the AF functionperformed in steps S71 to S78 in FIG. 14, the AF function can also beperformed to shift the focus position until the maximum value of thefocus score is detected, namely until the in-focus position is detected,for example.

In this case, in the AF function, the composite image can be generatedby using composition target images including a photographed imageobtained at the in-focus position and photographed images obtained at aplurality of focus positions preceding and following the in-focusposition.

<Fourth Configuration Example of Signal Processing Device 12>

FIG. 15 is a block diagram illustrating a fourth configuration exampleof the signal processing device 12 in FIG. 1.

Note that in the figure, a part corresponding to the one in FIG. 12 isassigned the same reference numeral as that in FIG. 12 to omitdescription of such part as appropriate.

The signal processing device 12 in FIG. 15 includes a frame buffer 31 toa control unit 35, a score calculation unit 71 to a peak detection unit74, and an AF frame setting unit 81.

Therefore, what is common to FIG. 12 in FIG. 15 is that the signalprocessing device 12 includes the frame buffer 31 to the control unit 35as well as the score calculation unit 71 to the peak detection unit 74.

The signal processing device 12 in FIG. 15 is however different fromthat in FIG. 12 in that the AF frame setting unit 81 is newly provided.

The signal processing device 12 in FIG. 15 has an AF function as is thecase with the example in FIG. 12.

However, an AF frame is set at a predetermined position such as thecenter of a photographed image in the example illustrated in FIG. 12whereas, in the example illustrated in FIG. 15, the signal processingdevice can set an AF frame at a position specified by a user in aphotographed image.

Specifically, the AF frame setting unit 81 sets the AF frame accordingto an AF mode and supplies it to the score calculation unit 71.

The score calculation unit 71 calculates a focus score by using thephotographed image within the AF frame supplied from the AF framesetting unit 81.

Here, the AF mode includes a normal mode and a specification mode.

The AF mode is set to the normal mode or the specification modeaccording to an operation of a user, for example.

In the normal mode, the AF frame setting unit 81 sets the AF frame at adefault position, namely at a predetermined position such as the centerof the photographed image.

In the specification mode, the AF frame setting unit 81 sets the AFframe at a position specified by the user on a photographed imageaccording to an operation of the user.

When the user specifies a position at an edge of the photographed image,for example, the AF frame setting unit 81 sets the AF frame at the edgeposition.

<AF Frame Setting Processing>

FIG. 16 is a flowchart illustrating an example of processing that setsthe AF frame in the signal processing device 12 of FIG. 15.

In step S91, the AF frame setting unit 81 determines whether a selectionoperation of selecting the AF mode is performed by the user.

When it is determined in step S91 that the selection operation ofselecting the AF mode is performed by the user, the operation proceedsto processing in step S92, in which the AF frame setting unit 81 sets(changes) the AF mode to the normal mode or specification mode accordingto the selection operation by the user, then the operation proceeds toprocessing in step S93.

On the other hand, the operation skips the processing in step S92 andproceeds to the processing in step S93 when it is determined in step S91that the selection operation of selecting the AF mode is not performedby the user.

In step S93, the AF frame setting unit 81 determines the (current) AFmode.

When it is determined in step S93 that the AF mode is set to the normalmode, the operation proceeds to processing in step S94, in which the AFframe setting unit 81 sets the AF frame at the center of thephotographed image or the like as a default position, then the operationreturns to the processing in step S91.

When it is determined in step S93 that the AF mode is set to thespecification mode, the operation proceeds to processing in step S95where the AF frame setting unit waits for the user to perform aspecification operation of specifying the AF frame and sets the AF frameat a position on the photographed image specified by the specificationoperation, then the operation returns to the processing in step S91.

Here, the specification operation can be performed by touching theposition on the photographed image displayed in a display device 13, forexample.

Note that the present technology can be applied not only to an imageobtained by photographing a human body but also an image obtained byphotographing a living body other than the human body.

The present technology can also be applied to an image obtained byphotographing an object other than the living body.

Moreover, the present technology can be applied to an arbitrary devicewith a photographing function other than medical equipment such as adigital camera, a vehicle-mounted image sensor, a surveillance camerainstalled for surveillance and security of agricultural products, and anindustrial endoscope (fiberscope) system.

Furthermore, the present technology can be applied to an imagephotographed by a device photographing visible light as well as an imagephotographed by a device photographing non-visible light(electromagnetic wave).

<Description of Computer to which Present Technology is Applied>

The series of processings performed by the signal processing device 12can be implemented by hardware or software. When the series ofprocessings are implemented by software, a program configuring thesoftware is installed on a general-purpose computer or the like.

Now, FIG. 17 illustrates a configuration example of an embodiment of acomputer installed with the program implementing the series ofprocessings.

The program can be recorded beforehand in a hard disk 105 or a ROM 103being a recording medium built into the computer.

Alternatively, the program can be stored (recorded) in a removablerecording medium 111. The removable recording medium 111 can then beprovided as so-called packaged software. Here, the removable recordingmedium 111 can be a flexible disk, a Compact Disc Read Only Memory(CD-ROM), a Magneto Optical (MO) disk, a Digital Versatile Disc (DVD), amagnetic disk, or a semiconductor memory, for example.

Note that the program can be installed on the computer from theremovable recording medium 111 or downloaded to the computer via acommunication network or broadcast network and installed on the built-inhard disk 105. Specifically, the program can be transmitted by radiofrom a download site to the computer via a satellite used for digitalsatellite broadcast or transmitted to the computer in a wired manner viaa network such as a Local Area Network (LAN) or the Internet, forexample.

A Central Processing Unit (CPU) 102 is incorporated into the computerand connected to an input/output interface 110 through a bus 101.

When a command is input by a user operation or the like on an input unit107 through the input/output interface 110, the CPU 102 runs a programstored in the Read Only Memory (ROM) 103 according to the command.Alternatively, the CPU 102 runs a program stored in the hard disk 105 byloading it to a Random Access Memory (RAM) 104.

The CPU 102 then performs the processing according to the aforementionedflowchart or the processing performed by the configuration illustratedin the aforementioned block diagram. Then, the CPU 102 for exampleoutputs or transmits the processing result from an output unit 106 or acommunication unit 108 through the input/output interface 110 andrecords the result in the hard disk 105 as needed.

Note that the input unit 107 is formed of a keyboard, a mouse, amicrophone and the like. The output unit 106 is formed of a LiquidCrystal Display (LCD), a speaker and the like.

Here, in the present specification, the processing performed by thecomputer according to the program does not have to be performed in timeseries in the order described in the flowchart. In other words, theprocessing performed by the computer according to the program includesprocessing performed in parallel or individually (such as parallelprocessing or processing by an object).

The program may be processed by a single computer (processor) orprocessed in a distributed manner by a plurality of computers. Theprogram may also be transferred to a remote computer and run.

Moreover, in the present specification, the system refers to theassembly of a plurality of components (such as devices and modules(parts)) where it does not matter whether or not all the components arehoused in the same housing. Accordingly, a plurality of devices housedin separate housings and connected through a network as well as a devicewith a plurality of modules housed in a single housing are both systems.

Note that the embodiments of the present technology are not limited tothe aforementioned embodiments, where various modifications can be madewithout departing from the scope of the present technology.

The present technology can for example employ cloud computing in which asingle function is shared and processed collaboratively among aplurality of devices through a network.

Moreover, each step described in the aforementioned flowcharts can beperformed by a single device or can be shared and performed by aplurality of devices.

When a single step includes a plurality of processings, the plurality ofprocessings included in the single step can be performed by a singledevice or can be shared and performed by a plurality of devices.

Furthermore, the effect described in the present specification isprovided by way of example and not by way of limitation, where there maybe another effect.

Note that the present technology can have the following configuration.

(1)

A medical image processing apparatus including:

-   -   circuitry configured to    -   generate a composite image by compositing a plurality of images        obtained by capturing with a medical imaging device a living        body while changing a focus position, and    -   switch output between the generated composite image and one of        the plurality of images based on a result of analysis performed        on at least one of the plurality of images.

(2)

The medical image processing apparatus according to (1), wherein thecircuitry is further configured to switch output between the generatedcomposite image and one of the plurality of images based on a manualinput operation.

(3)

The medical image processing apparatus according to (1)-(2), wherein theanalysis performed on at least one of the plurality of images includesanalysis of an angle of view adjustment.

(4)

The medical image processing apparatus according to (1)-(3), wherein theanalysis performed on at least one of the plurality of images includesmotion detection.

(5)

The medical image processing apparatus according to (1)-(4), wherein theanalysis performed on at least one of the plurality of images includesimage recognition that recognizes a treatment tool in the at least oneof the plurality of images.

(6)

The medical image processing apparatus according to (1)-(5), wherein thecircuitry is configured to switch the output between the generatedcomposite image and one of the plurality of images based on the resultof the analysis being lower than or equal to a predetermined threshold.

(7)

The medical image processing apparatus according to (2), wherein themanual input operation includes an input via an interface on a treatmenttool.

(8)

The medical image processing apparatus according to (1)-(7), wherein thecircuitry is configured to generate the composite image by selecting apixel of a previous composite image or a pixel of a newly captured imagethat is in focus and compositing the previous composite image and thelatest captured image.

(9)

The medical image processing apparatus according to (1)-(8), wherein themedical imaging device is configured to capture the plurality of imagesof the living body at an imaging frame rate, and

the circuitry is configured to output the generated composite image orthe one of the plurality of images at a display frame rate, the displayframe rate being equal to the imaging frame rate.

(10)

The medical image processing apparatus according to (1)-(9), wherein thecircuitry is configured to output the generated composite image or theone of the plurality of images at a display frame rate, the displayframe rate being at least 120 Hz.

(11)

The medical image processing apparatus according to (1)-(10), whereinthe circuitry is further configured to

perform alignment between the generated composite image and a newlycaptured image, and

update the composite image by compositing the aligned composite imageand newly captured image.

(12)

The medical image processing apparatus according to (11), wherein thecircuitry is configured to adjust an angle of view of each of thegenerated composite image and the newly captured image and performalignment between the generated composite image and the newly capturedimage after adjusting the angle of view.

(13)

The medical image processing apparatus according to (3)-(12), whereinthe circuitry is configured to switch output to the one of the pluralityof images when the analysis determines that a reliability of adjustmentof the angle of view performed on each of the generated composite imageand a newly captured image is lower than or equal to a threshold.

(14)

The medical image processing apparatus according to (11), wherein thecircuitry is configured to detect motion in each of the generatedcomposite image and the newly captured image and perform the alignmenton the basis of a result of the motion detection.

(15)

The medical image processing apparatus according to (4)-(14), whereinthe circuitry is configured to switch output to the one of the pluralityof images when the analysis determines that a reliability of the motiondetection is lower than or equal to a threshold.

(16)

The medical image processing apparatus according to (1)-(15), whereinthe plurality of images is obtained by capturing with the medicalimaging device the living body while changing the focus position withina range determined based on depth information of the living body.

(17)

The medical image processing apparatus according to (1)-(16), whereinthe circuitry is configured to generate the composite image bycompositing the plurality of images obtained at a focus position in apredetermined range including a peak position that is a focus positionat which a peak of a score, used in Auto Focus (AF), is obtained.

(18)

The medical image processing apparatus according to (1)-(17), whereinthe plurality of images includes a left eye image and a right eye image.

(19)

A medical image processing method including:

generating a composite image by compositing a plurality of imagesobtained by capturing with a medical imaging device a living body whilechanging a focus position; and

switching output between the generated composite image and one of theplurality of images based on a result of analysis performed on at leastone of the plurality of images.

(20)

A medical observation system including:

a medical imaging device configured to capture a plurality of images ofa living body while changing a focus position; and

circuitry configured to

generate a composite image by compositing the plurality of imagescaptured by the medical imaging device, and

switch output between the generated composite image and one of theplurality of images based on a result of analysis performed on at leastone of the plurality of images.

(21)

The medical image processing apparatus according to (20), wherein themedical imaging device is a surgical video microscope.

(22)

The medical image processing apparatus according to (20)-(21), whereinthe medical imaging device is an endoscope.

(23)

The medical image processing apparatus according to (20)-(22), whereinthe medical imaging device is configured to capture the plurality ofimages of the living body during a medical procedure.

(24)

A medical image processing apparatus including a composition unit thatgenerates a composite image by compositing a plurality of photographedimages obtained by photographing a living body while changing a focusposition, and generates a latest composite image by selecting a pixel ofa last composite image or a pixel of a latest photographed image that isin focus and compositing the last composite image and the latestphotographed image.

(25)

The medical image processing apparatus according to (24), wherein thecomposition unit outputs a photographed image as the composite imagewhen a composition restriction condition that restricts composition issatisfied.

(26)

The medical image processing apparatus according to (25), wherein thecomposition restriction condition is that a treatment tool photographedalong with the living body in the photographed image is in motion.

(27)

The medical image processing apparatus according to (25) or (26),further including an alignment unit that performs alignment between thecomposite image and the photographed image, wherein the composition unitcomposites the composite image and the photographed image that arealigned.

(28)

The medical image processing apparatus according to (27), wherein thealignment unit adjusts an angle of view of each of the composite imageand the photographed image and performs alignment between the compositeimage and the photographed image after adjusting the angle of view.

(29)

The medical image processing apparatus according to (28), wherein thecomposition restriction condition is that reliability of adjustment ofthe angle of view performed on each of the composite image and thephotographed image is lower than or equal to a threshold.

(30)

The medical image processing apparatus according to any of (27) to (29),wherein the alignment unit detects motion in each of the composite imageand the photographed image and performs the alignment on the basis of aresult of the motion detection.

(31)

The medical image processing apparatus according to (30), wherein thecomposition restriction condition is that reliability of the motiondetection is lower than or equal to a threshold.

(32)

The medical image processing apparatus according to any of (24) to (31),wherein a photographing unit that performs photographing to obtain thephotographed image obtains the photographed image while changing thefocus position within a range according to a user operation.

(33)

The medical image processing apparatus according to any of (24) to (31),wherein the composition unit composites a plurality of photographedimages obtained at a focus position in a predetermined range including apeak position that is a focus position at which a peak of a score usedin Auto Focus (AF) is obtained.

(34)

The medical image processing apparatus according to any of (24) to (33),wherein the photographing unit that performs photographing to obtain thephotographed image obtains a three-dimensional (3D) photographed image.

(35)

A medical image processing method including performing compositionprocessing that generates a composite image by compositing a pluralityof photographed images obtained by photographing a living body whilechanging a focus position, and generates a latest composite image byselecting a pixel of a last composite image or a pixel of a latestphotographed image that is in focus and compositing the last compositeimage and the latest photographed image.

(36)

A medical observation system including a photographing unit thatphotographs a living body while changing a focus position, and acomposition unit that generates a composite image by compositing aplurality of photographed images photographed by the photographing unitand generates a latest composite image by selecting a pixel of a lastcomposite image or a pixel of a latest photographed image that is infocus and compositing the last composite image and the latestphotographed image.

REFERENCE SIGNS LIST

-   -   11 Photographing unit    -   12 Signal processing device    -   13 Display device    -   21 Light source    -   22 Optical system    -   23 Image sensor    -   31 Frame buffer    -   32 Alignment unit    -   33 Composition unit    -   34 Drive control unit    -   35 Control unit    -   41 Angle-of-view adjustment unit    -   42 Motion blur elimination unit    -   43 Object alignment unit    -   51 Feature data calculation unit    -   52 Peak calculation unit    -   53 Image composition unit    -   61 Depth estimation unit    -   62 Range setting unit    -   63 Range storing unit    -   71 Score calculation unit    -   72 AF control unit    -   73 Buffer    -   74 Peak detection unit    -   81 AF frame setting unit    -   101 Bus    -   102 CPU    -   103 ROM    -   104 RAM    -   105 Hard disk    -   106 Output unit    -   107 Input unit    -   108 Communication unit    -   109 Drive    -   110 Input/output interface    -   111 Removable recording medium

1. A medical image processing apparatus comprising: circuitry configuredto generate a composite image by compositing a plurality of imagesobtained by capturing with a medical imaging device a living body whilechanging a focus position, and switch output between the generatedcomposite image and one of the plurality of images based on a result ofanalysis performed on at least one of the plurality of images.
 2. Themedical image processing apparatus according to claim 1, wherein thecircuitry is further configured to switch output between the generatedcomposite image and one of the plurality of images based on a manualinput operation.
 3. The medical image processing apparatus according toclaim 1, wherein the analysis performed on at least one of the pluralityof images includes analysis of an angle of view adjustment.
 4. Themedical image processing apparatus according to claim 1, wherein theanalysis performed on at least one of the plurality of images includesmotion detection.
 5. The medical image processing apparatus according toclaim 4, wherein the analysis performed on at least one of the pluralityof images includes image recognition that recognizes a treatment tool inthe at least one of the plurality of images.
 6. The medical imageprocessing apparatus according to claim 1, wherein the circuitry isconfigured to switch the output between the generated composite imageand one of the plurality of images based on the result of the analysisbeing lower than or equal to a predetermined threshold.
 7. The medicalimage processing apparatus according to claim 2, wherein the manualinput operation includes an input via an interface on a treatment tool.8. The medical image processing apparatus according to claim 1, whereinthe circuitry is configured to generate the composite image by selectinga pixel of a previous composite image or a pixel of a newly capturedimage that is in focus and compositing the previous composite image andthe latest captured image.
 9. The medical image processing apparatusaccording to claim 1, wherein the medical imaging device is configuredto capture the plurality of images of the living body at an imagingframe rate, and the circuitry is configured to output the generatedcomposite image or the one of the plurality of images at a display framerate, the display frame rate being equal to the imaging frame rate. 10.The medical image processing apparatus according to claim 1, wherein thecircuitry is configured to output the generated composite image or theone of the plurality of images at a display frame rate, the displayframe rate being at least 120 Hz.
 11. The medical image processingapparatus according to claim 1, wherein the circuitry is furtherconfigured to perform alignment between the generated composite imageand a newly captured image, and update the composite image bycompositing the aligned composite image and newly captured image. 12.The medical image processing apparatus according to claim 11, whereinthe circuitry is configured to adjust an angle of view of each of thegenerated composite image and the newly captured image and performalignment between the generated composite image and the newly capturedimage after adjusting the angle of view.
 13. The medical imageprocessing apparatus according to claim 3, wherein the circuitry isconfigured to switch output to the one of the plurality of images whenthe analysis determines that a reliability of adjustment of the angle ofview performed on each of the generated composite image and a newlycaptured image is lower than or equal to a threshold.
 14. The medicalimage processing apparatus according to claim 11, wherein the circuitryis configured to detect motion in each of the generated composite imageand the newly captured image and perform the alignment on the basis of aresult of the motion detection.
 15. The medical image processingapparatus according to claim 4, wherein the circuitry is configured toswitch output to the one of the plurality of images when the analysisdetermines that a reliability of the motion detection is lower than orequal to a threshold.
 16. The medical image processing apparatusaccording to claim 1, wherein the plurality of images is obtained bycapturing with the medical imaging device the living body while changingthe focus position within a range determined based on depth informationof the living body.
 17. The medical image processing apparatus accordingto claim 1, wherein the circuitry is configured to generate thecomposite image by compositing the plurality of images obtained at afocus position in a predetermined range including a peak position thatis a focus position at which a peak of a score, used in Auto Focus (AF),is obtained.
 18. The medical image processing apparatus according toclaim 1, wherein the plurality of images includes a left eye image and aright eye image.
 19. A medical image processing method comprising:generating a composite image by compositing a plurality of imagesobtained by capturing with a medical imaging device a living body whilechanging a focus position; and switching output between the generatedcomposite image and one of the plurality of images based on a result ofanalysis performed on at least one of the plurality of images.
 20. Amedical observation system comprising: a medical imaging deviceconfigured to capture a plurality of images of a living body whilechanging a focus position; and circuitry configured to generate acomposite image by compositing the plurality of images captured by themedical imaging device, and switch output between the generatedcomposite image and one of the plurality of images based on a result ofanalysis performed on at least one of the plurality of images.
 21. Themedical image processing apparatus according to claim 20, wherein themedical imaging device is a surgical video microscope.
 22. The medicalimage processing apparatus according to claim 20, wherein the medicalimaging device is an endoscope.
 23. The medical image processingapparatus according to claim 20, wherein the medical imaging device isconfigured to capture the plurality of images of the living body duringa medical procedure.